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Abstract 


This study presents an investigation of information literacy as defined by the ETS iSkills™ 
assessment and by the New Jersey Institute of Technology (NJIT) Information Literacy Scale 
(ILS). As two related but distinct measures, both iSkills and the ILS were used with 
undergraduate students at NJIT during the spring 2006 semester. Undergraduate students (n = 

331), first through senior years, took the iSkills and submitted portfolios to be judged by the ILS. 
First-year students took the Core iSkills assessment, which was designed to provide 
administrators and faculty with an understanding of the information and communication 
technology (ICT) literacy of a student doing entry-level coursework (n = 155). Upper classmen 
took the more difficult Advanced iSkills assessment, appropriate for rising juniors (n = 176). 
Across all class levels, iSkills scores varied as expected. First-year basic skills writing students 
performed at lower levels than first-year students enrolled in traditional composition and cultural 
history courses; seniors performed at higher levels than sophomores and juniors. Because the 
NJIT ILS scores were designed to be curriculum sensitive, portfolio scores did not similarly 
follow grade levels. Analyses revealed weak correlations between portfolio and Core iSkills 
scores and moderate correlations between portfolio and Advanced iSkills scores. As two 
associated yet distinct systems of inquiry designed to explore undergraduate student 
performance, the ETS iSkills assessment and the NJIT ILS—taken both individually and 
together—yield important infonnation regarding student performance. 

Key words: Information literacy, portfolio assessment, perfonnance assessment, information 
technology, writing assessment, construct validity 


l 



Acknowledgments 

The authors thank Lauren Molloy and Waverely VanWinkle for their help in conducting the 
literature review. Brent Bridgman, David Williamson, and Dan Eignor reviewed previous 
versions of this work. This research was funded by the ETS division of Higher Education and 
School Assessments and supported at NJIT by Richard Sweeney, Fadi Deek, and Robert Lynch. 


11 



Table of Contents 


Page 

Background: The ETS/ New Jersey Institute of Technology (NJIT) Collaboration.1 

Theoretical Grounding.2 

The Information and Communication Technology (ICT) Literacy Framework.2 

The NJIT Information Literacy Scale (ILS).7 

Method.11 

Participants.11 

Procedure.13 

Portfolio Scoring and Interrater Reliability.13 

Analyses.15 

Results.16 

Cohort Comparisons.16 

Correlations Among Variables.20 

Discussion.23 

The Construct of Information Literacy.23 

Concurrent Relationships.25 

Consequences of the Collaboration for NJIT.25 

References.28 

Appendix.33 


iii 





















List of Tables 


Table 1 

Table 2 
Table 3 

Table 4 
Table 5 


Interreader Reliability for NJIT Infonnation Literacy Scale (ILS): 

Nonadjudicated Scores. 

Means (Standard Deviations) for Two First-Year Cohorts. 

Means (Standard Deviations) for Three Groups Taking the Advanced Level 

iSkills Assessment. 

Intercorrelations for First-Year Students. 

Intercorrelations for Sophomores, Juniors, and Seniors. 


Page 

... 14 
... 18 

... 18 
...21 
...22 








Background: The ETS/ New Jersey Institute of Technology (NJIT) Collaboration 

In the fall of 2004, librarians and faculty at New Jersey Institute of Technology (NJIT) 
began a fonnal investigation of the information literacy skills of undergraduate students. Working 
with specialists in research and infonnation literacy at the university’s Robert Van Houten Library, 
instructors in the department of humanities worked to design an infonnation literacy model based 
on standards derived from the Association of College & Research Libraries (ACRL). In that the 
faculty had been assessing the writing skills of students enrolled in general undergraduate 
requirements (GUR) in humanities since 1996, a traditional portfolio assessment system had 
emerged that allowed reliable and valid programmatic infonnation to be gained about student 
writing (Elliot, Briller, & Joshi, 2007). A new portfolio assessment system launched in spring 
2005—tenned the NJIT Infonnation Literacy Scale (ILS)—shifted the assessment focus from 
writing to information literacy assessment (Scharf, Elliot, Huey, Briller, & Joshi, 2007). While 
allowing similarly strong validity evidence to be wananted as the original portfolio system, the 
information literacy scores were lower than anticipated. Instructional and library faculty were 
interested in learning more about the information literacy skills of their students. 

In fall 2005, NJIT and ETS undertook a collaborative research agreement to investigate 
more fully—by means of multiple approaches—the variables of information literacy as they were 
evidenced within student performance at a public comprehensive technological university. The 
collaboration was designed to provide insight into the following questions: 

• What kind of validity evidence based on the construct of infonnation literacy could be 
warranted from an analysis of the student scores on the ETS iSkills™ assessment and 
the NJIT ILS? While the concept of the ETS iSkills assessment focuses on infonnation 
and communication technology (ICT) literacy, the NJIT ILS focuses on the kinds of 
information literacy skills needed for proficiency in writing research papers in the 
humanities. The relationship between the concepts of the two assessments—both 
critical in nature yet differently articulated within a higher education environment—is 
unknown. 

• What kinds of evidence could be warranted based on the relationship of the two 
measures of information literacy to other variables? If the ETS iSkills assessment and 
the NJIT ILS posited associations that were congruent yet distinct, as we hypothesized, 
the relationships between these measures and general academic measures (course grade, 
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grade point average [GPA], and scores on the College Board’s SAT® Math [SAT-M] 
and SAT Verbal [SAT-V] assessments) refine our understanding of the discrimination 
between ICT literacy and NJIT’s concept of information literacy in the humanities. 

• Based on the consequences of the release of the scores and the consequences of the 
collaboration itself, what kinds of evidence could be warranted to describe the impact 
of the ETS and NJIT collaboration upon the NJIT community? While the impact of 
various kinds of portfolio scoring at NJIT had been demonstrated—both internally to 
the institution (e.g., New Jersey Institute of Technology, 2007, pp. 53-54) and 
externally to a national community (e.g., Coppola & Elliot, 2007)—the impact of an 
assessment system using a nationally developed assessment of information literacy and 
a locally developed assessment of that construct is unknown. 

The American Educational Research Association, the American Psychological Association, 
and the National Council on Measurement in Education (1999) provide a meaningful heuristic to 
the process of validation in Standards for Educational and Psychological Testing. By reflecting on 
the construct at hand, examining the relationship of the construct to other variables, and 
documenting the consequences of the assessment activity, we offer the following case study as a 
heuristic by which the concept of information literacy may be more fully understood. 

Theoretical Grounding 

The Information and Communication Technology (ICT) Literacy Framework 

Predating the collaboration described in this case study, ETS convened an international 
panel in 2001 to study current and emerging information and communication technologies and 
their relationship to critical cognitive skills (International ICT Literacy Panel, 2002). 
Understanding that information and communication technologies cannot be defined as the mastery 
of technical skills, the international panel concluded that the cognitive skills involved in 
information literacy included general literacy (reading and numeracy), critical thinking, and 
problem solving. Subsequent to identifying a need for a measure of ICT literacy, the panel worked 
to develop a framework for that literacy. A consortium of seven college and university systems 
worked with ETS to tailor this ICT literacy framework to the needs of higher education, refining 
the intended construct in the process. Over a 2-year period, consortium members and other 
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institutions collaborated in the design, development, and testing of the iSkills assessment of ICT 
literacy. 

Conceptual definition of ICT literacy. The definition of ICT literacy adopted by the 
consortium members reflects this comprehensive view of information literacy: 

ICT literacy is the ability to appropriately use digital technology, communication tools, 
and/or networks to solve information problems in order to function in an information 
society. This includes having the ability to use information as a tool to research, organize, 
and communicate information and having a fundamental understanding of the ethical/legal 
issues surrounding accessing and using information. (Katz et ah, 2004, p. 7) 

Content domain of the iSkills assessment. Consortium members further refined and 
deepened this construct, identifying seven perfonnance areas (Katz, 2007): definition (using ICT 
tools to identify and appropriately represent an information need), access (collecting and retrieving 
information in digital environments), evaluation (determining the degree to which digital 
information satisfies the needs of the task in ICT environments), management (applying an 
existing organizational or classification scheme for digital infonnation), integration (interpreting 
and representing digital information), creation (generating infonnation by adapting, applying, 
designing, or inventing information in ICT environments), and communication (sharing 
information properly in its contexts of use for ICT environments). 

Research regarding the skills required to use ICT tools to define an information need and to 
access information has revealed that both disciplinary expertise and general search expertise affect 
how a person searches for information. Actions taken by both domain and search experts in 
accessing information are associated with efficient and productive patterns. Experts take less time 
and fewer actions to locate relevant information, for instance, and less time to complete 
successfully the information-seeking task (Downing, Moore, & Brown, 2005; Lazonder, Beimans, 
& Wopereis, 2000). Search experts are more likely than novices to look over results before moving 
on (Hsieh-Yee, 1993) and choose target documents for closer inspection (Holscher & Strube, 
2000). The search refinement strategies of domain and Web novices, conversely, are associated 
with patterns of inefficiency, and these search patterns are often redundant with only minor 
strategic changes made from one query to the next (Hembrooke, Granka, Gay, & Liddy, 2005; 
Holscher & Strube, 2000). In addition, novice searchers waste significantly more time navigating 
and backtracking than do experts (Holscher & Strube, 2000; Luconi & Tabatabai, 1999; Tabatabai 
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& Shore, 2005). Search expertise and domain knowledge each make a unique contribution to the 
ways in which people seek information. 

Much research on information literacy has focused attention on the ways and extent to 
which information is evaluated. A number of factors may be considered in infonnation evaluation, 
including trustworthiness, relevance, currency, accuracy, objectivity, sufficiency, resource type, 
and ethical use. Empirical studies cited by Rieh (2002) demonstrate that cognitive authority and 
information quality are important standards of judgment that experts use in evaluating the 
usefulness of information. Interviews, verbal protocols during searches, and search logs of a small 
yet diverse group of scholars documented the consideration of trustworthiness, reliability, 
scholarliness, credibility, and authority when judging infonnation sources (Rieh, 2002). 

Studies of novice users, however, reveal a lack of such judgmental standards. In a survey of 
1,050 college students who use the Internet, only 4% of respondents felt that the quality of the 
information they found through Web searches was not of sufficient quality for their school 
assignments; half of all students surveyed, however, believed strongly that the infonnation on the 
Web was indeed acceptable for school assignments. Almost two-thirds of students believed that 
the range of resources on the Web was adequate for their needs (OCLC Online Computer Library 
Center, 2002). Confinning such a lack of judgmental standards, Hepworth (1999) posited that only 
minimal student knowledge existed regarding the differing value of various sources of infonnation. 

Managing abundant information is often noted as a challenge in today's infonnation-rich 
society. The field of human-computer interaction is a recognized source of empirical research 
regarding the application of existing organizational schemes on artifacts of digitally transmitted 
information such as e-mails and electronic files. Research regarding the strategies by which 
professionals and students save e-mail folders, for example, reveals that such digital information, 
once archived into the computer for a long time, is rarely archived out (Boardman & Sasse, 2004; 
Lantz, 1998). Such high volume of e-mails subsequently leads to excess clutter that makes e-mail 
difficult to manage, organize, and locate (Boardman & Sasse, 2004; Lantz, 1998). In a relational 
study, Lantz identified a significant correlation (0.72) between the number of messages that 
employees store in their inbox and problems managing e-mail. Research on information 
management behavior can also provide insight into infonnation management patterns. Bondarenko 
and Janssen (2005) concluded that documents related to administrative activities are easier to 
categorize than those related to research. While administrative activities that have a precise 
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function (e.g., preparing for a meeting at the end of the week) remain relevant for a much shorter 
period of time, research documents encompass more variety and thus remain active for a longer 
period of time. Human-computer interaction studies have much to contribute to an understanding 
of storage and organizational patterns within ICT environments. 

As a skill needed for information literacy, integration has received little empirical attention. 
Integration, an act of bringing together potentially disparate pieces of information from different 
sources containing different information formats, is the mediating function between creation and 
communication. Insight into the little studied but complex nature of integrative acts is seen in a 
study by Jones, Bruce, and Dumais (2001) on the shortcomings of bookmarking as a means of 
storing information. While two tools provided by the Web browser (history and bookmarking) 
were among the least frequently used methods for saving Web information, other methods were 
found to be more logical choices for integrating information. Techniques such as printing the Web 
page guarantees persistence of information, strategies such as saving the Web page itself as a file 
guarantees preservation of information in its current state, and simply pasting the Web address into 
an e-mail or word processing document with a short description allows for establishment of a 
context. Such varied practices allow integration into an existing organizational scheme and yield 
ease in sharing information. 

The literature of composition studies reveals much about the creation and communication 
of information. Sheehy (2003) examined two competitive forces in an analysis of student work. 
Students were instructed to present an argument to the school board against closing their school, 
and the researcher collected audio-taped classroom and group discussions, student writing, and the 
texts from which students drew their infonnation. In her 8-week study, Sheehy found that that the 
students were informed by their perceived knowledge of audience, even though they received 
standardized instructions of task assignment given by the teacher. Reviewing this research, Beach 
and Friedrich (2006) noted that the study demonstrates the importance of perceptions about 
audience knowledge and beliefs in fulfilling communicative tasks. As Nelson (2008) has noted, 
social cognition—the ability to consider perspectives of others and to make assumptions about 
what they will gain from reading—is key to the communicative process (Bonk, 1990; Rubin, 

1984). When Kroll (1984) asked participants of various ages to revise their texts to serve the needs 
of particular audiences, he identified distinct differences in the ability of older participants to 
reduce the level of lexical and syntactic complexity for young readers. It appears, as Nelson 
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suggests, that developmental differences in social cognition may be evidenced in the ability to 
adapt a text to a given audience (Kroll, 1984, p. 439). Such research supports the iSkills 
assessment’s inclusion of such tasks as creating presentation slides to support a position (an act of 
creation) and transforming a document to make it more useful to a particular group (an act of 
communication). 

Administration of the iSkills assessment. The ETS iSkills assessment is an Internet- 
delivered assessment. In that the assessment focuses on cognitive problem-solving and critical 
thinking skills associated with using technology to handle information, the scoring algorithms 
target cognitive decision making rather than technical competencies. Assessment administration 
takes approximately 75 minutes, divided into two sections lasting 35 and 40 minutes, respectively. 
During this time, students respond to 15 interactive tasks that are performance-based. Each 
interactive task presents a real-world scenario, such as a class or work assignment, that frames the 
information task. Students solve the tasks in the context of a simulation (for example, e-mail, Web 
browser, or library database) having the look and feel of typical applications. In the assessment, for 
example, students might encounter a scenario requiring the use of a search engine to access 
information from a database. The results are tracked and strategies scored based on how well the 
students search for infonnation, such as key words chosen and refinement of search strategies, and 
how well the infonnation returned meets the demands of the task. As a scenario-based assessment, 
students become engaged in the world of the tasks—themselves representative of the types of ICT 
literacy assignments students should be seeing in their coursework. As a simulation, task-based 
assessment, the iSkills assessment purports to elicit higher-order critical thinking and problem¬ 
solving skills. Katz (2007) provided further details on the assessment, including its development 
and field testing. 

The iSkills assessment has two versions, Core and Advanced. The Core iSkills assessment 
was designed for students entering higher education, such as college freshmen. The Advanced 
iSkills assessment was designed for students moving to upper level coursework, such as 
sophomores and juniors. Identical in structure, general content, and assessment approach, the Core 
and Advanced assessment tasks differ in their complexity. Core tasks were designed to be easier, 
with lower reading loads, more straightforward task instructions, and fewer options than Advanced 
tasks. In the present study, first-year students took the Core iSkills assessment and second year and 
higher students took the Advanced iSkills assessment. 
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The NJIT Information Literacy Scale (ILS) 

It is the social and cognitive aspects of written communication, reflecting critical thinking 
and problem-solving ability, that are under investigation in the student portfolios required by the 
NJIT Department of Humanities. Indeed, these portfolios serve as the vehicles that capture the 
information literacy skills of NJIT undergraduate students as that ability is socially formed and 
cognitively tasked to exhibit critical reflection and problem exposition within courses. 

The literature of composition studies provides the research background for the definition of 
information literacy employed by NJIT. In her social cognitive theory of writing, Flower (1994) 
emphasized a model in which readers and writers construct meaning within a three-fold context of 
social and cultural information, of language processes, and of discourse conventions. Such a broad, 
outer circle of influence is informed by a more immediate circle of general purposes, specific 
goals, and activated knowledge linked to the task at hand. These public and situational contexts 
must also be understood as intertwined with an individual’s personal cognitive constructs, those 
individualized contours of thought that shape the act of meaning making. The mental 
representations—the webs of meaning that writers construct—reveal the complex interactions of 
forces often in conflict. As well, Flower identifies individual metacognitive awareness through 
which individuals may, or may not, be aware of their own constructive processes. For Flower, 
meaning is created and communicated through reproduction, conversation, and communication. In 
her interviews and protocol analyses of college-age writers, Flower focuses on ways to teach the 
strategic acquisition of knowledge. Her research informs our understanding of the complex social 
and cognitive processes that influence the construction of literate acts. Methods for analyzing 
students’ texts, such as rhetorical structure theory (RSA; Mann & Thompson, 1988), and 
procedures for incremental structure analysis (PSIA; Sanders & Van Wijk, 1996) also hold 
promise to reveal detailed information about the cognitive dimensions of writing (Sanders & 
Schilperood, 2006). 

Conceptual definition of writing. Within the NJIT Department of Humanities, we have 
adopted a definition of writing given by Camp (1996): “a rich, multifaceted, meaning-making 
activity that occurs over time and in a social context, an activity that varies with purpose, situation, 
and audience and is improved by reflection on the written product and on the strategies used in 
creating it” (p. 135). We have expanded that definition as follows: 
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Writing is an act of critical thinking that extends beyond summary into persuasion, an act 
that improves with drafting and is best when informed by the voices of others, an act that is 
interrelated with demands of informal and fonnal oral presentation and that is often 
undertaken in collaboration with fellow writers. (Elliot et ah, 2007) 

Conceptual definition of information literacy. This definition of writing is taken as the 
construct domain that has been assessed at NJIT since 1996. The definition of information literacy, 
understood as a set of behaviors included as part of the construct of writing, was formed by 
librarians and instructional faculty in spring 2005: 

Within the Department of Humanities at NJIT, information literacy is the ability to 
demonstrate that a coherent, planned intellectual framework has been used to identify, find, 
understand, and use infonnation in drafting, revising, and finalizing researched, persuasive 
writing (Scharf et ah, 2007). 

Content domain of the NJIT ILS. The NJIT definition of infonnation literacy was and 
remains based on the definition of information literacy offered by the Middle States Commission 
on Higher Education (MSCHE; Middle States Commission on Higher Education, 2006). Because 
the portfolios exist within a specific institutional site and because the GUR impacted the entire 
undergraduate curriculum at NJIT, instructors wanted to address competency issues associated 
with accreditation. Addressing the goals of information literacy as established by the ACRL and 
embraced by the MSCHE was of paramount importance. To meet such demands, librarians and 
faculty members further refined and deepened this construct, identifying four performance areas to 
be used in the assessment of infonnation literacy within student portfolios: citation (the ability to 
cite sources so that information may be identified without undue burden), evidence of independent 
research (the use of information beyond the syllabus), appropriateness (the use of sources germane 
to the topic), and integration (the use of sources marbled into the development of ideas). These 
four component variables thus became the NJIT ILS. As component variables, these four 
characteristics of information literacy were associated with an overall information literacy 
portfolio score (a holistic impression of research competency). 

In previous assessment work by humanities faculty at NJIT, the citation variable was 
judged as the ability of students to properly cite their sources according to the style of the Modern 
Language Association or the American Psychological Association. The meaning of this component 
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variable was expanded for information literacy study. Because citing sources so they could be 
found was more important than strict adherence to a standard citation style, evidence of a student’s 
sense of a retrievable reference suggested that the student understood the particular attributes of a 
source. Such evidence addressed the Information Literacy Competency Standards for Higher 
Education developed by the ACRL (2002) as Performance Outcomes 2.5, c and d. Competence 
would be exhibited if students differentiated between types of sources and included all pertinent 
information in the varying cases so that sources could be retrieved by a reader without undue 
burden. 

Regarding the importance of independent research, NJIT instructional faculty felt that it 
was important that student papers provided evidence that research had been conducted extending 
beyond the syllabus (and sources) recommended by the instructor. Papers with little variety of 
sources in scope, subject, and format were unlikely to have been well researched by the student. 
Conversely, if the student sought ideas from a variety of additional sources to become truly 
informed about the topic at hand, evidence would be present that ACRL Standards 1 and 2 were 
being addressed. 

To investigate appropriateness, NJIT instructors sought to determine if students chose 
sources that were not only relevant, but that also had a high probability of being accurate and 
authoritative. If so, the students would be addressing ACRL Standards 1, 3, and 4 and the 
standards’ emphasis that infonnation-literate students should be required to evaluate information 
and sources critically and to incorporate selected infonnation into an information base and value 
system. 

With the most complex variable—that of integration—instructional faculty sought to 
determine if sources were truly integrated into the portfolio papers. To judge the work against 
ACRL Standards 3 and 4, the readers were asked to evaluate the student’s arguments and ideas. 
Realizing that a citation may be merely cosmetic in nature, faculty sought evidence of integration 
that included the use of concepts from outside sources to build a foundation and compare, contrast, 
and refute arguments. The use of in-text citations was thus linked to concepts and arguments used 
to provide further evidence. Faculty sought to investigate the degree to which a student was able 
not only to summarize the main ideas from sources consulted (ACRL Performance Indicator 3.1), 
but also to synthesize ideas to construct new concepts (ACRL Performance Indicator 3.3). To meet 
Standard 4—to use “information effectively to accomplish a specific purpose”—the sources cited 
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had to have been used reflectively in the paper. If a student was able to use outside information as 
part of the knowledge base on which the writing in the portfolio was developed, it would be clear 
that ACRL Performance Indicator 4.1 was being addressed. 

Administration of the NJITILS. Unlike the detailed, time-intensive methods undertaken in 
RSA and PSIA analysis, the NJIT effort is designed to allow all instructors teaching undergraduate 
courses to come together each semester and, within less than 3 hours, reliably evaluate the work of 
a representative number of students by the methods described below. 

The NJIT ILS assessment is used within a program assessment environment. Within a 
planned sequence of assessment episodes each semester, evidence is gathered about the 
performance of students in different undergraduate cohorts. The spring 2006 assessment described 
in this study was devoted to the assessment of information literacy by the two methods herein 
described. Because the individual instructor’s grade is not an influence in the assessment 
process—indeed, portfolio scoring takes place after final grades are given—the NJIT assessment 
program avoids the consequential issues encountered within an environment of high-stakes testing. 
Assessments always take place face-to-face, usually in a morning or afternoon. In that the majority 
of the instructors are employed full-time, the assessment environment is always collegial; all 
involved realize that they are present to learn more about the abilities of their students, critical 
shareholders whom they will very likely encounter again in subsequent semesters. 

During the assessment period, trained instructors evaluate student portfolios, artifacts 
capturing the work completed in a 15-week semester. Within the portfolio are contained a variety 
of documents, depending on the cohort and instructor: annotated planning bibliographies, 
proposals for research projects, drafts of various writing tasks, evidence of collaborative work, and 
researched final documents. With the courses in technical writing, the documents may be 
contained in a student-designed Web site, the site itself designed according to audience-based 
usability principles. Common to all aspects of the undergraduate curriculum is the emphasis on 
persuasion. While no claim can be made that the Toulmin (2003) model is universally adopted, 
various strategies of warranting evidence are common elements across the NJIT undergraduate 
humanities curriculum. 

Differences in the iSkills and the ILS assessments. While the NJIT ILS was informed by a 
literature review similar to that which informed the ETS ICT literacy framework, the purpose of 
the NJIT assessment differs in two ways from the iSkills assessment. First, within the specific 
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institutional site, humanities instructors were interested in the variables of information literacy as 
they were articulated within the undergraduate curriculum. As such, the NJIT ILS focused on 
written products—the researched, persuasive documents contained in portfolios. Unlike the iSkills 
assessment, the NJIT assessment does not therefore account for the process by which students 
completed their classroom writing assignments. Second, because students take humanities GUR 
courses from first through senior years, instructors realized that they were capable of investigating 
differences within levels—but not across levels—of the offered curriculum. Instructors realized 
authentic and comprehensive assessments—“ideally constructed to examine how much students 
have grown during the college experience”—were needed (Ewell, 2006). Yet because course 
content differed from the first to the senior years, the NJIT assessment focused on the context in 
which the portfolios emerged; hence the portfolio scores were not designed to follow grade level 
but, instead, were designed to reveal performance of students in the humanities classes in which 
they were enrolled. 

The present case study of NJIT undergraduate students thus allowed investigation of the 
way that students defined, accessed, managed, integrated, evaluated, created, and communicated 
information in the broad context of information literacy, a task that is often (though not always) 
executed within a set time frame reflective of the 75-minute assessment. As well, the case study 
allowed investigation into the effectiveness with which students cited sources, launched 
independent research, employed appropriate sources, and integrated their ideas with the ideas of 
others, a task that is often (though not always) executed within a set time frame reflective of the 
duration of a semester course. In some instances, we hypothesized we would identify congruence 
between the two assessment methods; in other instances, valuable discriminant evidence on the 
different aspects of information literacy captured by the two measures would be identified. Such 
evidence could be used in planning future instruction and assessment of NJIT undergraduate 
students. 


Method 

Participants 

Late March, April, and early May of 2006 provided an ideal occasion for the administration 
of the iSkills assessment. Planning the assessment to follow a traditional withdrawal date in the 
third week of March, the researchers believed that the NJIT students remaining in undergraduate 
classes were those most committed to successful completion of the class. In addition, the five 


11 



student cohorts of the study—writing students (first year), cultural history students (first year), 
cultural history (second and third year), technical writing students (third and fourth year), and 
senior seminar students (third and fourth year)—allowed a comprehensive representation of NJIT 
undergraduate students. Because students from each of the cohorts were required to submit 
portfolios at the end of the semester to their instructors, students whose portfolios were targeted for 
submission would also take the appropriate version of the iSkills assessment. 

A simple random sample of students was created across each section of all but the senior 
seminars (for which all students were selected as described below), and students were identified 
for portfolio submission. Students who were unable or declined to take the iSkills assessment were 
replaced with other randomly selected students (who, in turn, were then identified for portfolio 
submission). Students who did not submit portfolios for any reason or who submitted portfolios 
with no work suitable for the NJIT ILS rubric (i.e., no researched documents) were excluded from 
analyses. The senior seminar students consisted of a census of all whose transcripts revealed that 
they had never taken any course outside of NJIT; hence, these students, while small in number, 
represented a meaningful population of NJIT students. 

First-year students were found in first-year writing courses (n =110) and in cultural history 
courses (n = 45). These first-year students all took the Core iSkills assessment. Students taking the 
Advanced iSkills assessment were also found in cultural history (n = 95), as well as in technical 
writing (n = 48) and the senior seminars (n = 33). 

The sampling plan design closely matched the NJIT student population. For example, 
among the first-year students for whom SAT scores were available, the overall average SAT-M 
and SAT-V scores for the entire first-year class (SAT-M = 588, SAT-V = 525) were comparable to 
that of the tested population (SAT-M = 578, SAT-V = 509). The gender of the first-year sample 
(85% male) was similarly representative of all first-year NJIT students (83% male). The first-year 
sample was also comparable to all first-year NJIT students on the basis of ethnicity: 13% versus 
11% African-American students, 35% versus 37% Caucasian students, and 16% versus 15% 
Hispanic students. Asian students were slightly overrepresented in the first-year sample (23% 
versus 18% overall). The remaining students comprise those of other races or those who declined 
to answer the ethnicity question. 
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Procedure 

In late March 2006, researchers contacted and provided instructors in each of the four 
cohorts with a list of their students who were to take the ETS iSkills assessment and submit 
portfolios for the NJIT ILS. A member of the NJIT research team then met the students in their 
classroom and escorted them to secure computers in the NJIT Robert Van Houten Library. To 
ensure the integrity of iSkills assessment scores, research team members proctored the test-taking 
sessions. 

In May 2006, portfolios of targeted students enrolled in first-year writing, cultural history, 
and the senior seminars were evaluated according to each of the variables in the NJIT ILS. 

Because students in the technical writing courses were evaluated by means of variables related to 
that course’s content (Johnson, 2006), only two NJIT ILS variables were evaluated in that cohort: 
citation and evidence of research. 

Portfolio Scoring and Interrater Reliability 

While the administration of the iSkills assessment required careful planning, scoring was 
computer based and, thus, efficient. While execution of the sampling plan for the NJIT portfolio 
assessment required a level of planning equal to that of the iSkills administration, scoring of the 
portfolios with the NJIT ILS was complex and time consuming, requiring 3- to 4-hour sessions with 
all instructors for the first-year writing, cultural history, technical writing, and senior seminar 
courses. Following procedures established for portfolio assessment of writing ability at NJIT (Elliot 
et ah, 2007), sample portfolios served as models to orient readers to the information literacy 
assessment process. The 75- to 90-minute orientation included independent scoring by readers of the 
samples, followed by group discussion to help readers calibrate their assessments and agree about 
the parameters that would ensure consistency. A reading session followed the orientation. 

The combination of the scoring sheet (see the appendix) and the sample training portfolios 
was used so readers would score according to the functional perfonnance level expressed in the 
scoring sheet as well as calibrate performance within the range of student sample portfolios. 

Two readers independently read and evaluated each portfolio using the infonnation literacy 
scoring sheet, and steps were taken to make sure that the readers did not know each other’s scores. 

In addition, none of the instructors read their own students’ portfolios. Each of the five portfolio 
scores was expressed as the total of two readers’ scores. The first four judgments (see the 
appendix) represent components of information literacy (citation, evidence of research, 
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appropriateness, and integration); the fifth is an overall assessment of the portfolio. Readers were 
encouraged to judge each variable independently of their judgments of other variables. The overall 
score, therefore, was not a component score but a holistic judgment (White, 1984) that may, or 
may not, relate to scores given for other, analytically read variables (Purves, Gorman, & Takala, 
1988). Table 1 shows the interrater agreement for these scores using weighted kappa with 
quadratic weights (as is appropriate for ordered categories). Following the strength of agreement 
benchmarks described by Landis and Koch (1977), the ratings are in moderate to substantial 
agreement for first-year writing, cultural history, and the senior seminars, while the technical 
writing scores may be classified as being in fair agreement. 


Table 1 

Interreader Reliability for NJIT Information Literacy Scale (ILS): Nonadjudicated Scores 


Variables 

lst-year 
writing 
(n = 110) 

Cultural 
history 
(1 st -year 
students; 
n = 45) 

Cultural 

history 

(2nd-3rd year 
students; 
n = 95) 

Technical 
writing 
(3rd-4th year 
students; 
n = 48) 

Senior 
seminar 
(3rd-4th year 
students; 
n = 33) 

ILS components 
Citation 

.601 

.734 

.623 

.386 

.535 

Evidence of 
research 

.588 

.639 

.734 

.276 

.484 

Appropriateness 

.459 

.496 

.711 

n/a a 

.525 

Integration 

.501 

.546 

.681 

n/a a 

.603 

Holistic ILS 

.660 

.956 

.755 

n/a a 

.599 


Note. Scores were adjudicated before used in analyses. See the Portfolio Scoring and Interrater 
Reliability subsection in this report for a description of the adjudication procedure. 
a A weighted kappa was not run for the technical writing students because no appropriateness, 
integration, or holistic ILS score was derived in the study. 


Following the writing assessment model, the information literacy assessment model held 
that any score on any of the four component variables or on the overall (holistic) information 
literacy portfolio score would have to be adjudicated by a third reader if the first two readers did 
not award matching or adjacent scores. Thus, a portfolio receiving a score of 5 (indicating that the 
first reader strongly agreed with the statement) and a score of 3 (indicating that the second reader 
disagreed with the statement) would be sent to a third reader who would then make an independent 
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judgment and resolve the discrepancy. For consistency, in cases where a third reading could be 
resolved in either direction (e.g., Reader 1 = 4, Reader 2 = 2, Reader 3 = 3), then the higher 
summed score (7) would be awarded. 

Analyses 

The goal of the analyses was to investigate the similarities and differences in measurement 
provided by the iSkills assessment and NJIT portfolio rubric. Data were analyzed from three 
sources: iSkills scores, portfolio scores, and NJIT student records. The latter included academic 
variables such as SAT scores, GPA, and grades in the humanities courses (in which students 
submitted portfolios) for the spring 2006 semester. Analyses included comparison of mean scores 
across the five student groups and inspection of correlations from the combined sample. The 
variables included in the analyses are as follows: 

• iSkills scores. Scores on the iSkills assessment range from 0 to 300 for the Core iSkills 
assessment and from 400 to 700 on the Advanced iSkills assessment. Reliability 
(Cronbach alpha) for both tests is approximately 0.80. 

• Holistic ILS. The overall assessment of a portfolio is rated on a scale from 1 (very 
strongly disagree that the contents of the portfolio demonstrate that the student has 
employed an information literacy framework) to 6 (very strongly agree). The variable 
used in these analyses is the sum, ranging from 2 to 12, of the adjudicated ratings of 
two judges. Technical writing students’ portfolios were not assessed holistically, so 
these students did not receive a holistic ILS score. 

• Component ILS. The ILS includes four component scales (citation, application, 
evidence, and integration; Questions 1 to 4, respectively in the appendix), which are 
each rated from 1 to 6 (very strongly disagree to very strongly agree with the rubric’s 
analytic statements). The score for each component is the sum of the adjudicated 
ratings, ranging from 2 to 12, from two judges,. To simplify discussion, analyses used 
the mean of the four component scores for each student. Cronbach alpha for the four- 
item scale is 0.87. Note that the technical writing students were scored on a 
modification of the ILS that included only the citation and application scales. For this 
group, Cronbach alpha for the two-item scale is 0.77. 
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• Course grade. This variable is a numerical translation of the letter grade each student 
received in the humanities class from which he or she was recruited. The variable 
ranges from 0 (F) to 4 (A). Course withdrawals and incompletes were interpreted as 
missing data. 

• GPA. This variable is the undergraduate GPA of each student, including all courses up 
to the semester in which the study occurred. As with course grade, values range from 0 
to 4. 

• SAT-Mand SAT-V. These SAT-M and SAT-V scores were obtained from NJIT student 
records. Each score ranges from 200 to 800. Cronbach alpha for SAT-M and SAT-V 
have been reported as 0.92 and 0.93, respectively (Ewing, Huff, Andrews, & King, 
2005). 


Results 

The results are organized as follows. First, we compare cohorts in their performance on the 
two information literacy variables (iSkills and ILS) and other academic variables, analyzing the 
students who took the Core iSkills assessment separately from those who took the Advanced 
version. Next, we inspect the correlations among these variables, again separately considering the 
two groups of freshmen from the three groups of upper classmen. Conducting the analyses in this 
way allows us to consider the differencing perfonnance with respect to the level of iSkills 
assessment taken (Core or Advanced). 

Cohort Comparisons 

At NJIT, first-year students are placed into either a 2-semester basic skills writing course 
(Reading, Writing, Speaking I and II) or a 1-semester writing course (Writing, Speaking, 
Thinking). Because the speaking component has proven complex to assess, current efforts 
concentrate solely on the writing component of the courses. Basic writers track into the 2-semester 
sequence because of low scores on the SAT-V and on locally developed placement tests. Students 
from both tracks were sampled during the spring semester to take the Core iSkills assessment. One 
group was sampled from Reading, Writing, Speaking II to represent basic writers. The second 
group was enrolled in cultural history. First-year students in this course (a) needed no remediation 
in writing, (b) had taken the prerequisite 1-semester writing course (Writing, Speaking, Thinking) 
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during the fall semester, (c) received a passing grade, and (d) immediately enrolled in cultural 
history, the next course in the NJIT humanities sequence. Such students, as the experience of 
instructors reveals, show confidence in their ability to perform well in humanities courses and a 
desire to move forward in the course sequence. 

Comparisons between these two groups illuminated differences in the performance of basic 
versus nonbasic writers on information literacy measures. Because SAT-V scores were used to 
determine tracking, it is not surprising that these groups differ on this variable (first-year writing: 

M = 487, SD = 70; cultural history: M = 562, SD = 61), although they have similar SAT-M scores 
(first-year writing: M = 572, SD = 66; cultural history: M = 594, SD = 68), GPAs (first-year 
writing: M = 2.8, SD = 0.7; cultural history: M = 2.9, SD = 0.6), and grades in their respective 
courses (first-year writing: M = 3.1, SD = 0.9; cultural history: M = 3.2, SD = 0.7). 

Table 2 shows the means and standard deviations for the infonnation literacy variables. 
Preliminary analyses revealed that SAT-V score was a significant covariate for Core iSkills scores 
but not for the ILS scores. The comparison on iSkills scores used analysis of covariance to control 
for the effect of SAT-V scores. The comparisons on holistic and component ILS involved checking 
on differences in means via a /-test. On all variables, first-year writing students scored lower, 
although only scores on component ILS were statistically significant. Hence, the lower scores of 
this group reflect the presence of basic writers. Indeed, as a portfolio score of 7 is considered the 
lowest acceptable by NJIT instructional faculty, the holistic ILS Score (6.7) indicates that students 
barely meet expectations. While the holistic ILS scores of the cultural history students did not 
differ significantly from the scores of those enrolled in first-year writing, it is nevertheless 
important to note that the score did indicate an acceptable level of proficiency. The significantly 
higher scores of the cultural history students on the component ILS demonstrate, again, that these 
students possess a higher level of information literacy skill. Among all of the five groups in the 
case study, only the first-year cultural history students and the senior seminar students (see Table 
3) submitted portfolios that demonstrated competency—a score at or above 7—on both the holistic 
ILS and component ILS variables. 

Table 3 provides the means and standard deviations of the information literacy variables for 
the three cohorts that took the Advanced iSkills assessment. 
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Table 2 

Means (Standard Deviations) for Two First-Year Cohorts 



Lirst-year 
writing 
(n =110) 

Cultural 
history 
(n = 45) 

Statistic ( df) 

p 

iSkills scores (Core) 

133.7(35.2) 

157.9 (31.3) 

C(l,152) = 1.3 

ns 

Holistic ILS 

6.7 (1.9) 

7.4 (2.4) 

"5n 

oo 

II 

NO 

ns 

Component ILS 

7.1 (1.7) 

7.9 (2.1) 

(N 

II 

57 

NO 

*0 

<.05 


Note. Three students did not have SAT scores in their NJIT student records. T-tests for holistic and 
component ILS scores assumed unequal variances between the independent groups. Effect size for 
component ILS analysis is d= 0.42. 


Table 3 


Means (Standard Deviations) for Three Groups Taking the Advanced Level iSkills Assessment 


Cultural history 
(n = 95) 

Technical 
writing 
(n = 48) 

Senior seminar 
(n = 33) 

F(2,143) 

P 

2 

Partial rf 

iSkills scores 
(Advanced) 

548.5a (36.9) 

547.2a (39.5) 

568.3b (28.2) 

3.9 

<.05 

0.05 

Holistic ILS 

6.8 (2.4) 

n/a 

7.0 (2.2) 


ns d 


Component ILS 

7.4 a (2.1) 

6.2b (2.0) 

7.4 a (1.7) 

4.90 

< .01 

0.06 

Course grade 

3.2 (0.8) 

3.4 (0.8) 

3.5 (0.5) 

1.90 

ns 


GPA 

2.9 a (0.5) 

2.9 a (0.6) 

3.1 a (0.4) 

3.30 

< .05 

0.04 

SAT-M 

585.4 (73.6) 

571.5 (88.1) 

610.3 (67.5) 

2.20 

ns 


SAT-V 

511.5 (81.8) 

513.0(98.7) 

528.4 (40.1) 

0.55 

ns 



Note. Different subscripts within a row represent means different at the 0.05 level by Tukey’s 
honestly significant difference test. Partial rfi is an effect size measure representing the proportion 
of the total variance attributed to an effect. GPA = grade point average, SAT-M = SAT Math, 


SAT-V = SAT Verbal. 

a Because the technical writing group did not have holistic ILS scores, a /-test was used to compare 
the cultural history and senior seminar groups on this variable (?[125]=.38, ns). 
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NJIT students may take cultural history at any time after the Writing, Speaking, Thinking 
course (or after the 2-semester basic writing sequence). While some first-year students move 
immediately into cultural history after fulfilling the first-year writing requirement, others delay 
taking the course until their sophomore or junior year. These sophomores and juniors represented 
one cohort that took the Advanced iSkills assessment. Students within the second cohort taking the 
Advanced iSkills assessment, those enrolled in the technical writing course, were required to fulfill 
only the first-year writing requirement as a prerequisite. Students enrolled in the senior seminars, 
the most advanced level of students taking the Advanced iSkills assessment, must have 
successfully completed the demanding humanities courses required of all NJIT undergraduate 
majors: the first-year writing requirement; the cultural history requirement; the basic social science 
requirement; the literature, history, and philosophy or the science, technology, and society 
requirement; and the open elective in humanities or social science. Although the iSkills Advanced 
assessment is not necessarily intended for graduating students, the NJIT researchers wanted to 
evaluate the performance of the university’s highest level of student. In that many of NJIT’s 
students either transfer into the curriculum or take their lower-division humanities requirements at 
other post-secondary institutions, the researchers wanted to gain a better understanding of the 
information literacy abilities of students who had taken all of their undergraduate humanities and 
social science work at NJIT. Students who had taken any English as a second language course 
were excluded as well. Of the 287 students enrolled after the March withdrawal deadline in spring 
2006, only 33 students met these inclusion criteria. 

Students enrolled in cultural history who were not first-year students earned lower scores in 
both the holistic ILS (M = 6.8, SD = 2.4) and the component ILS (M = 7.4, SD = 2.1; Table 3) 
than did their first-year classmates (Table 2). However, the cultural history students’ iSkills 
Advanced scores (M = 548.5, SD = 36.9) were comparable with those of students enrolled in 
technical writing (M = 547.2, SD = 39.5). In that both cultural history and technical writing have 
the same prerequisite of the first-year writing course, the nearly identical iSkills Advanced scores 
of both groups suggest that additional information literacy instruction may not be forthcoming 
from other coursework outside of humanities courses. 

Distressing to the NJIT instructional staff was the absence of evidence that students in 
technical writing had gained proficiency in the areas of citation and evidence of independent 
research, the two variables examined in this cohort of student portfolios. The component ILS score 
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for these two combined variables (M = 6.2, SD = 2.0) is significantly lower than the scores for 
either the cultural history (M = 7.4, SD =2.1) or the senior seminar students (M = 7.4, SD = 1.7). 
The technical writing students’ component ILS (composed of only citation and evidence of 
independent research; M = 6.2, SD = 2.0) is also lower than the mean of the two corresponding 
scores for cultural history (M = 7.8, SD = 2.1) and senior seminar students (M = 7.7, SD = 1.6). 

Less distressing, however, were the scores of the senior seminar students. Scharf, Elliot, et 
al. (2007), investigating a similar group of students at NJIT (n = 100), found holistic ILS scores 
below the cut score of 7 (M = 6.14, SD = 2.9; Scharf, Elliot, et al., 2007, p. 648), as were the 
scores of each of the other variables that make up the component ILS variable used in the current 
study. One year later, in the current study, both the holistic ILS scores (M = 7.0, SD = 2.2) and the 
component ILS scores (M = 7.4, SD = 1.7) of a similar cohort were higher, although just barely 
meeting expectations. It is important, nevertheless, to note that such scores were lower than the 
instructors were accustomed to seeing in the variables associated with the traditional NJIT writing 
assessment undertaken in the senior seminar, as reported by Scharf, Elliot, et al. (2007, p. 648): 
Critical Thinking (M = 8.94, SD = 1.46), Drafting (M = 7.73, SD = 2.65), Citation (M = 7.45, SD 
= 2.61), and Overall Writing Score (M = 8.89, SD = 1.5). As expected, the students in the senior 
seminar demonstrated significantly higher iSkills Advanced scores than students enrolled in 
cultural history and technical writing. 

Correlations Among Variables 

The intercorrelations among the variables are shown in Table 4 for the first-year students 
(those taking the Core iSkills assessment). 

These analyses suggest distinctions between the ILS and iSkills measures. Information 
Literacy Scale scores are more closely correlated with each other (r = .90), course grade, and GPA 
than with the other variables. These correlations are higher than those observed 1 year earlier by 
Scharf, Elliot, et al. (2007). Lor example, Scharf, Elliot, et al. reported a correlation of .28 (p. 470) 
between holistic ILS scores and course grade, while the corresponding correlation here is .41. 
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Table 4 


Intercorrelations for First-Year Students 



iSkills 

scores 

(Core) 

Holistic 

ILS 

Component 

ILS 

Course 

grade 

GPA 

SAT-M 

SAT-V 

iSkills scores (Core) 

- 

0.18* 

0.12 

0.11 

0.08 

0.33** 

0.52** 

Holistic ILS 


- 

0.90** 

0.41** 

0.30** 

0.17* 

0.10 

Component ILS 



- 

0.42** 

0.33** 

0.14 

0.11 

Course grade 




- 

0.62** 

0.09 

0.08 

GPA 





- 

0.09 

0.06 

SAT-M 






- 

0.28** 

SAT-V 







- 


Note: ILS = Information Literacy Scale, GPA = grade point average, SAT-M = SAT Math, SAT-V 
= SAT Verbal. 

*p < .05. **p < .01. 


Regarding general academic performance, although the portfolio scores are correlated with 
GPA (holistic ILS, r = .30, component ILS, r = .33), the alignment with SAT scores is 
considerably weaker. The weak correlations between grades (both course grade and GPA) and 
SAT scores might be due to differing grading standards between the basic writing course and 
cultural history. For basic writing students, grades may take into account factors such as evidence 
of reasoning, resilience, and responsibility (Sternberg & Subotnik, 2006)—all elements of 
classroom instruction that instructors seek to nurture within a fragile population. Conversely, the 
cultural history students who took the iSkills Core are among the most tenacious students, those 
who proceed directly from first-year writing to the next required humanities course. As such, these 
more resilient students are more likely to be evaluated on a content-based grading system that is 
uniformly traditional. Indeed, within just the cultural history course, correlations are higher 
between SAT scores and both course grades (SAT-V: r = .33; SAT-M: r = .22) and GPA (SAT-V: 
r = .24; SAT-M: r = .54) than they are for the full group of first-year students. For the first-year 
writing students, the corresponding correlations are all near zero (ranging from -.08 to .05). As 
Rose (1988) has famously pointed out, basic writers employ complex cognitive ability that is not 
uniform in its application to writing contexts. As such, the classroom assessment framework is not 
likely to be as uniform as it is for those students who are stronger writers—such as the first-year 
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cultural history students—and thus adapt to writing assignments using more uniform writing 
strategies, techniques that are more unifonnly evaluated in a final grade. 

The correlations of chief interest in this study—between iSkills scores and portfolio 
scores—are weak, with little to suggest comparable measurement. The only significant correlation 
identified is between the iSkills Core and the holistic ILS (r = . 18). Any connection between 
iSkills Core scores and portfolio scores appears to be due to general academic ability. Note that 
there are significant correlations between both the iSkills Core and the SAT-M and SAT-V scores 
of first-year students. Indeed, the high correlation between the iSkills Core and the SAT-V 
suggests that verbally oriented critical thinking skills may be common to both assessments. Partial 
correlations controlling for SAT-M and SAT-V suggest the lack of unique correspondence 
between iSkills and portfolio scores: holistic ILS-iSkills: p = .12; component ILS-iSkills: p = .06. 

The intercorrelations among the variables are shown in Table 5 for the sophomore, junior, 
and senior students (those taking the Advanced iSkills assessment). 


Table 5 


Intercorrelations for Sophomores, Juniors, and Seniors 



iSkills 

scores 

(Advanced) 

Holistic 

ILS 

Component 

ILS 

Course 

grade 

GPA 

SAT-M 

SAT-V 

iSkills scores 
(Advanced) 


0.23** 

0.21** 

0.21** 

0.27** 

0.38** 

0.49** 

Holistic ILS 


- 

0.93** 

0.35** 

0.25** 

0.05 

0.25** 

Component ILS 
Course grade 
GPA 

SAT-M 

SAT-V 




0.37** 

0.25** 

0.54** 

0.08 

0.20* 

0.32** 

0.17* 

0.37** 

0.41** 

0.52** 


Note: ILS = Infonnation Learning Scale; GPA = grade point average. 


*p < .05. **p < .01. 


Compared with the results for the freshmen, these data show stronger relationships among 
most measures. Again, there is strong correlation between the holistic and component scores on the 
ILS (r = .93), a pattern maintained by Scharf, Elliot, et al. (2007). As with first-year students, the 
correlation between holistic ILS and course grade increased from the Scharf, Elliot, et al. study (r 
= .28) to the present study conducted 1 year later (r = .35). The portfolio scores are similarly 
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correlated with course grades and GPA. In addition, while no correlation was evident between the 
ILS scores and the SAT-V scores in the first-year study, ILS scores are more strongly correlated 
within the group of upper classmen (holistic ILS, r = .25, component ILS, r = .17). The iSkills 
Advanced scores are moderately correlated with course grade (r = .21) and GPA (r = .27), the 
latter at a similar level to other research on the iSkills assessment (Katz & Smith-Macklin, 2007). 
As was the case for the first-year students, iSkills correlated well with SAT scores, in particular 
with SAT-V scores (r = .49), as befits a measure of information handling skills. Finally, moderate 
correlations exist between iSkills and portfolio scores (holistic ILS, r = .23, component ILS, r = 
.21). However, as was the case for the first-year students, these correlations are moderated by 
students’ general academic skills as measured by SAT scores. Partial correlations controlling for 
SAT-M and SAT-V are quite low: holistic ILS-iSkills: p = .14; component ILS-iSkills: p = .15. 

Discussion 

An empirical investigation at NJIT began in fall 2004 when instructors recognized that the 
information literacy skills of their students were wanting. In fall 2007, the time of the present 
writing, NJIT has established and evaluated the information literacy skills of their students by two 
related yet distinct measures: the ETS iSkills assessment and the NJIT ILS assessment. What has 
the institution gained in its ability to warrant validity evidence based on its assessment of the 
information literacy construct? What has the institution gained in its ability to provide evidence 
based on the relationship of the two assessments to other variables of academic interest? And what, 
in terms of consequence, has the institution gained in terms of the impact of the two assessment 
systems? 

The Construct of Information Literacy 

The iSkills Core assessment correlated weakly with the holistic ILS, although the 
correlations with both holistic and component ILS improved somewhat in the iSkills Advanced 
assessment. For both versions of the iSkills assessment, SAT-V and SAT-M scores appeared to 
moderate most of the relationship with portfolio scores. The ETS and NJIT assessments appear to 
be more distinct than related. 

Yet these results are quite meaningful, a reflection of two appropriately different 
definitions and measures of information literacy. Attention to the variable of integration, present 
on both assessments, suggests a possible reason for the low correlations. Within the ETS 
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framework, integration stresses the interpretation and representation of information from multiple 
sources. An integrative task could ask students to compare advertisements from competing 
vendors’ Web sites by summarizing information into a table, or students could be asked to 
rerepresent results from a sporting event into a spreadsheet to clarify standings and decide the need 
for playoffs. Within the NJIT framework, the demand for integration also asks students to cull 
information from multiple sources to facilitate critical thought. However, as the appendix shows, 
the aim is to interpret, deepen, and reflect on a topic within a written document. As it is articulated 
in the fifteen-week cultural history courses, students engaging in an integrative task would be 
asked to read Mary Shelly’s Frankenstein and think about the underlying assumption in Western 
thought that the world is mechanistic and that there is a decreasing role of divinity. Students would 
be encouraged to go to the library for hard copy books, such as Leo Marx’s The Machine in the 
Garden: Technology and the Pastoral Idea in America, and to use the library databases, such as 
JSTOR, to identify peer-reviewed articles such as Barbara Johnson’s “My Monster/My Self’ from 
a 1982 archived issue of Diacritics. Students would prepare a series of drafted and edited 
documents, perhaps accompanied by an oral presentation. The final research paper might argue 
that the relationship between parenthood and monstrousness is examined by Shelly within a 
fatalist, deterministic setting in which divinity is absent but technology is ever present. Such 
activities are, in fact, common with our instructors (Hetherington, 2007). 

The process of such critical reflections is the vehicle by which the concept of information 
literacy is operationalized within the NJIT humanities framework. The content domain of 
information literacy, intermixed with highly demanding reading and persuasive writing tasks, is 
executed in a different time frame and with an approach distinct from that used by the ETS iSkills 
assessment. The tasks and the constructs both assessments embody may be related, yet they are 
nevertheless distinct. Of course, infonnation literacy as mediated by a humanities-oriented 
framework for writing may itself be distinct from the goal-directed writing of specific disciplines. 
Indeed, recent work suggests a relationship between iSkills assessment scores and grades in a 
business writing course (Katz, Haras, & Blaszczynski, 2008). 

Without question, NJIT nevertheless values both constructs. The humanities tasks fit within 
the mission of that academic unit, just as the iSkills tasks apply across the curriculum. Students 
need to write and reflect during extended periods, just as they need to evaluate rapidly much of the 
information they encounter daily. Indeed, it is interesting that the correlations between the iSkills 
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scores and the holistic and component ILS scores are higher in sophomore, junior, and senior 
students, as compared with first-year students. Perhaps students gain both sets of cognitive 
complex abilities—those demanded by the iSkills and ILS tasks—as they progress through the 
curriculum. If so, then that congruence of domain and task would be an ideal academic outcome: 
an integration of discriminate skills, related yet distinct, required for all graduates. 

Concurrent Relationships 

The correlation of chief interest was the relationship between the iSkills scores and the 
portfolio scores, and the distinct nature of the two assessments was demonstrated. Evidence of 
concurrent validity, shown in Tables 4 and 5, is found in the consistent relationship of the iSkills 
Core and Advanced assessments to the SAT-M and SAT-V scores. In addition, scores on the 
iSkills Advanced assessment were all correlated with academic criterion measures (course grade 
and GPA). These results are consistent with the idea that some commonality is present among the 
information literacy skills of these more developed writers (as evaluated by products in portfolios 
and grades in their humanities classes), their more general infonnation literacy processes (as 
evaluated by the iSkills Advanced assessment), and general academic ability (as revealed in GPA 
and SAT scores). 

The ILS, conversely, demonstrates little or no relationship with SAT-M or SAT-V 
measures. Instead, both the holistic and component ILS scores correlate with course grades and 
GPA for all students, reflecting the appropriately curricular-bound emphasis of the ILS. Because 
the ILS was intended to measure the information literacy skills of students as they are manifested 
within written documents prepared for specific classes, the value of such an assessment is 
strengthened when it is shown to be related to academic performance at each grade level. 

Consequences of the Collaboration for NJIT 

Evaluation of the consequences of assessment must be part of all program evaluation 
(Messick, 1994). Instead of considering consequences a factor apart from the investigation of 
construct and concurrent relationships, the consequences involved with the assessment of 
information literacy should be warranted as equally important to the success of the assessment. 
Along with the gains realized through a highly articulated model of information literacy and 
empirical assessment of student ability, NJIT has realized a more fully articulated sense of 
information literacy as administrators and instructors have begun, in committee and classroom, to 
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address the information literacy skills of students. Even the rater agreement measures noted in this 
case study may be understood as evidence of the capability of faculty and librarians to unite in 
pursuit of a common assessment goal involving a new, yet critical, literacy that is as important to 
student success as academic writing ability—higher education’s so-called composition emphasis— 
was to students at the turn of the 20th century. 

In preparing The Future’s Edge (New Jersey Institute of Technology, 2007), a periodic 
accreditation review report prepared for the MSCHE, the office of the president featured both the 
traditional writing assessment of portfolios conducted by the department of humanities (pp. 46-47) 
and the new collaborative information literacy assessment with ETS (pp. 53-54). Demonstration of 
efforts to assess information literacy within the undergraduate population was clearly important to 
the NJIT administration, and research with ETS allowed NJIT to follow Category 8 
Characteristics of Programs of Information Literacy That Illustrate Best Practices: A Guideline in 
its suggestion that multiple methods for program evaluation are needed for effective outcomes 
assessment (Association of College & Research Libraries, 2003). On November 27, 2007, the 
NJIT report was accepted and reaccreditation was confirmed, with the assessment of information 
literacy now a part of the university’s narrative of success. Concurrent with the preparation of the 
accreditation report, NJIT began a Provost’s task force on information, communication, and 
technology literacy and a Web site at the Robert Van Houten Library hosting research and 
curricular advances. 

In addition to university-wide efforts, NJIT has added a second composition course— 
English Composition: Writing, Speaking, Thinking II—focusing on methods of research using 
print and online sources, the evaluation and proper citation of source material, and the 
development of researched essays and report writing. In spring 2008, 12 sections of the course are 
planned to accommodate 288 students. While the librarians will maintain their close relationship 
with the humanities, they have begun to address information literacy instruction and assessment in 
departments across the university. Informed by the results of the present study, the librarians 
would be the first to counsel that students who are selected to take the iSkills assessment must be 
identified from various departments within the academic institution; information literacy must not 
be seen to reside solely within the domain of the humanities. 

External to NJIT, preliminary results of the present study were presented at the ETS iSkills 
National Advisory Committee (Scharf et ah, 2006). As well, a replication study of the ILS was 
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undertaken with the University of North Carolina, Charlotte (Scharf, Huey, Elliot, Metzger, & 
Gunter, 2007). The infonnation literacy effort at NJIT has become part of the NJIT recruitment 
plan, with an advertisement in the New York Times Education Life supplement noting information 
literacy among the skills offered at NJIT to give students a competitive edge in their fields of study 
(“Envision Your Future,” 2007, p. 4). A subsequent front page article in The Star Ledger, 
referencing NJIT’s efforts with ETS, further deepened the environment of infonnation literacy at 
NJIT (Heyboer, 2007). 

Best understood as related yet distinct measures, the NJIT ILS and the ETS iSkills 
assessment together provide a fuller construct representation of infonnation literacy for the 
university than either measure separately. While it is clear from this case study that there is much 
room for improvement regarding the information literacy abilities of NJIT undergraduate students, 
the across-the-cuniculum orientation gained by employing both measures has resulted in an 
instructional emphasis that is already demonstrating gains in awareness of the important and 
diverse nature of infonnation literacy skills within a technological research university. Indeed, in 
that infonnation literacy may be understood as an emerging construct—a point made by Tyler 
(2005) in associating information literacy with emerging global competitiveness—it is heartening 
to see the content of both the NJIT and ETS assessments so readily confirmed by instructors and 
librarians over the past 3 years. Significantly, at NJIT the definition of literacy as both an 
individual and communal good is strengthened by the use of both assessment systems, a 
consequence that has obviated the value dualisms often associated with literacy (Brandt, 2004). 

The collaborative effort described in this report has provided a combined assessment for New 
Jersey’s only comprehensive technological university. While future studies are planned, they will 
be possible only because of the unique collaborative research model—one that recognizes the 
hannony that can and should exist between discriminant measures of infonnation literacy— 
described in this report. 
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Appendix 

NJIT ILS Portfolio Scoring Sheet 


The NJIT ILS portfolio scoring sheet is from The Direct Assessment of Information 
Literacy at NJIT: A Portfolio Assessment Model by N. Elliot, D. Scharf, H. A. Huey, V. Briller, 
and K. Joshi, 2006, retrieved February 11, 2008, from the Information Literacy at NJIT Web site: 
http://libraryl.njit.edu/infolit/researchreport.pdf Copyright 2006 by New Jersey Institute of 
Technology. Used with pennission. 
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NJIT Assessment Scales: Information Literacy 


Reader’s Name: 

Date: 

Student’s Name: 

Course: 


The Middle States Commission on Higher Education defines information literacy as “an intellectual 
framework for identifying, finding, understanding, evaluating and using information. It includes determining 
the nature and extent of needed information; accessing information effectively and efficiently; evaluating 
critically information and its sources; incorporating selected information in the learner’s knowledge base and 
value system; using information effectively to accomplish a specific purpose; understanding the economic, 
legal and social issues surrounding the use of information and information technology; and observing laws, 
regulations, and institutional policies related to the access and use of information.” It is the presence and 
extent of such literacy that we are assessing as it exists within undergraduate courses offered by the 
Department of Humanities at NJIT. 

1. Citation : This portfolio includes sources that are documented so that the original source 
can easily be found. Discussion: All information needed to identify a source must be present. The 
audience-centered ability of students to present a source that may be retrieved without undue burden is more 
important than stylistic adherence to a particular citation system. 

The contents of the portfolio demonstrate that the student has cited sources so that the original source 
can be easily found. _____ 


Very Strongly 

Strongly Agree 

Agree 

Disagree 

Strongly 

Very Strongly 

Agree 




Disagree 

Disagree 


2. Evidence of Independent Research : This portfolio includes evidence of research 
independent of sources indicated within the course syllabus. Discussion: While it is important that 
students reference information from textbooks, readers, and bibliographies provided by the instructor, 
researched work demands that students have sought, evaluated, and used information beyond the syllabus. An 
authentically researched assignment demonstrates that the student has sought ideas from a variety of sources 
to become truly informed about the topic at hand. 

The contents of the portfolio demonstrate that the student has performed independent research. 


Very Strongly 

Strongly Agree 

Agree 

Disagree 

Strongly 

Very Strongly 

Agree 




Disagree 

Disagree 


3. Appropriateness : The sources used in this portfolio are appropriate to the topic the student 
addressed. Discussion: Academic integrity demands that authoritative sources must be used in researched 
work. Research that is appropriate to the topic at hand will be sensitive to issues such as validity, timeliness, 
and sufficiency’. An authentically researched assignment will demonstrate a student’s ability to identify valid 
sources that have been reliably reviewed by those recognized as knowledgeable about the topic at hand, to 
select sources that offer time-appropriate views on that topic, and to ensure that the sources used are adequate 
to support the demands of the topic. 

The contents of the portfolio demonstrate that the student has used appropriate sources. _ 


Very Strongly 

Strongly Agree 

Agree 

Disagree 

Strongly 

Very Strongly 

Agree 




Disagree 

Disagree 


4. Integration : The sources cited in this portfolio have informed the course work. Discussion: 
Authentically researched work will demonstrate that the student has incorporated information in order to 
deepen critical thought. Authentic integration will demonstrate that the student has used sources to interpret, 
deepen, and reflect on the topic at hand. 

The contents of the portfolio demonstrate that the student has integrated sources. _ 


Very Strongly 

Strongly Agree 

Agree 

Disagree 

Strongly 

Very Strongly 

Agree 




Disagree 

Disagree 


5. Overall Information Literacy Portfolio Score : 

The contents of the portfolio demonstrate that the student has employed an information literacy 
framework. 


Very Strongly 
Agree 

Strongly Agree 

Agree 

Disagree 

Strongly 

Disagree 

Very Strongly 
Disagree 

The materials in the 
portfolio 

demonstrate superio 
information literacy 
skills. 

The materials in the 
portfolio 
demonstrate very 
good information 
literacy skills. 

The materials in the 
portfolio demonstrate 
and an acceptable leve: 
of information literacy 
skills. 

The materials in the 
portfolio demonstrati 
below average 
information literacy 
skills. 

The materials in the 
portfolio demonstrati 
information literacy 
skills at a level near 
failure. 

The materials in the 
portfolio demonstrate 
information literacy 
skills at a level of 
failure. 
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